Repository navigation
chdb: one long-lived server process instead of a Python process per query - #3444
alexey-milovidov wants to merge 4 commits into
Conversation
…uery `./start` now runs `chdb/server.py`: a single Python process with one chDB instance over `.clickbench`, serving queries on a Unix socket. Every connection is served by its own thread with its own `Session`, reused for all queries on that connection. `./query`, `./check`, `./load` and the new `./concurrent-worker` are stdlib-only clients (`chdb/client.py`). Before, every `./query` opened a `Session` on `.clickbench` in a fresh process, so concurrent queries failed with "Cannot lock file .clickbench/status" (3/300 successful queries with 4 workers in the randomized benchmark group a73b3f6e-b715-480c-b46a-2cb2fd0dbc09). The entry is now restartable (a cold run restarts the server), and the concurrent QPS test is enabled, with one connection (one session) per worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl)
|
Results for Logs:
|
|
Results for Logs:
|
|
Results for Logs:
|
neuroalexey
left a comment
There was a problem hiding this comment.
Status: Please verify — Q5 on c6a.large is 9 times slower
chdb/results/20261119/c6a.large.json:20 has Q5 (COUNT(DISTINCT UserID)) at 20.1 to 21.7 sec in all three runs. The previous result has 2.2 to 2.5 sec. All 11 earlier c6a.large results are under 5.2 sec.
t3a.small has half the memory, but it runs Q5 in 5.2 sec here. Is it expected?
I ran main's validator on this head. It reports nothing for chdb.
Automated review by neuroalexey
./startnow runschdb/server.py: one long-lived Python process with a single chDB instance over.clickbench, accepting concurrent connections on a Unix socket (chdb.sock). Every connection is served by its own thread with its ownSession, reused for all queries sent over that connection:./query,./check,./loadand the new./concurrent-workerare stdlib-only clients (chdb/client.py), so they don't load chDB. The time is measured by the server aroundsession.query.Motivation: before, every
./querystarted a new Python process and opened aSessionon.clickbench, so concurrent queries failed withCannot lock file .clickbench/status. In the randomized benchmark groupa73b3f6e-b715-480c-b46a-2cb2fd0dbc09(4 workers), chdb had 3/300 successful queries.Other changes in the entry:
BENCH_RESTARTABLEis now the defaultyes: a cold run restarts the server.BENCH_CONCURRENT_WORKER=./concurrent-workerkeeps one connection, so one session, per worker, and reconnects if the watchdog restarts the server../loadgoes through the server (DROP TABLE IF EXISTS clickbench.hits SYNC, thencreate.sqlandinsert.sql), because the server holds the data directory.Tested locally with chdb 4.5.1 / chdb-core 26.11.1 on a 2M-row
generateRandomtable: 8 parallel./querycalls all succeeded; 4 concurrent workers ran ~950-990 queries each in 20 s with 0 errors; data survives a stop and start; a warm startup takes 0.3-0.45 s. A full benchmark run has not been done yet.🤖 Generated with Claude Code